Distribution and storage of data on local and remote disks in multi-use clusters of commodity PCs

Over the last few decades, the power of personal computers (PCs) has grown steadily, following the exponential growth rate predicted by Moore’s law. The trend towards the commoditization of PC components (such as CPUs, memories, high-speed interconnects and disks) results in a highly attractive price/performance ratio of the systems built from those components. Following these trends, I propose to integrate the commodity IT resources of an entire company or organziation into multi-use clusters of commodity PCs. These include compute farms, experimental clusters as well as desktop PCs in offices and labs. This thesis follows a bottom-up architectural approach and deals with hardware and system-software architecture with a tight focus on performance and efficiency. In contrast, the Grid view of providing services instead of hardware for storage and computation deals mostly with problems of capability, service and security rather than performance and modelling thereof. Multi-use clusters of commodity PCs have by far enough storage on their hard-disk drives for the required local operating-system (OS) installation and therfore there is a lot of excess storage in a multi-use cluster. This additional disk space on the nodes should be put to a better use for a variety of interesting applications e.g. for on-line analytic data processing (OLAP). The specific contributions of the thesis include solutions to four important problems of optimized resource usage in multi-use-cluster environments. Analytic models of computer systems are important to understand the performance of current systems and to predict the performance of future systems early in the design stage. The thesis instroduces a simple analytic model of data streams in clusters. The model considers the topology of data streams as well as the limitations of the edges and nodes. It also takes into account the limitations of the resources within the nodes, which are passed through by the data streams. Using the model, the thesis evaluates different data-casting techniques that can be used to replicate OS installations to many nodes in clusters. The different implementations based on IP multicast, star-, treeand multi-drop–chain topologies are evaluated with the analytic model as well as with experimental measurements. As a result of the evaluation, the multi-drop chain is proposed as most suitable replication technique. When working with multi-use clusters, we noticed that maintenance of the highly replicated system software is difficult, because there are many OS installations in different versions and customisations. Since it is desirable to backup all older versions and customisations of all OS installations, I implemented several techniques to archive the large amounts of highly redundant data contained in the nodes’ OS partitions. The techniques take different approaches of comparing the data, but are all OS independent and work with whole partition images. The block repositories